Combining 3D Image and Tabular Data via the Dynamic Affine Feature Map Transform

Prior work on diagnosing Alzheimer’s disease from magnetic resonance images of the brain established that convolutional neural networks (CNNs) can leverage high-dimensional image information for classifying patients. However, little research focused how these models utilize usually low-dimensional tabular information, such as patient demographics or laboratory measurements. We introduce Dynamic Affine Feature Map Transform (DAFT), a general-purpose module CNNs dynamically rescales and shifts feature maps layer, conditional patient’s clinical information. show DAFT is highly effective in combining 3D diagnosis time-to-dementia prediction, where it outperforms competing with mean balanced accuracy 0.622 c-index 0.748, respectively. Our extensive ablation study provides valuable insights into architectural properties DAFT. implementation available at https://github.com/ai-med/DAFT.